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1.  EXECUTIVE  SUMMARY 


The  University  of  Wyoming  has  formed  a  robotics  initiative  consisting  of  three  distinct  parts.  A 
complete,  stand-alone  final  technical  report  is  presented  for  each  phase.  Phase  1  was  managed  by 
Dr.  Cameron  Wright,  Phase  2  by  Dr.  John  O’Brien,  and  Phase  3  by  Dr.  John  Mclnroy.  The 
overall  project  was  coordinated  by  the  Robotics  Initiative  Manager,  Dr.  John  Mclnroy. 

Phase  1,  “Biomimetic  Vision  Sensor,”  AFRL-RX-TY-TR-201 1-0096-01  summarizes  the 
development  of  a  novel  computer  vision  sensor  based  upon  the  biological  vision  system  of  the 
common  housefly,  Musca  domestica.  Several  variations  of  this  sensor  were  designed,  simulated 
extensively,  and  hardware  prototypes  were  constructed  and  tested.  Initial  results  indicate  much 
greater  sensitivity  to  object  motion  than  traditional  sensors,  and  the  promise  of  very  high  speed 
extraction  of  key  image  features.  The  main  contributions  of  this  research  include:  (1) 
characterization  of  the  image  information  content  presented  by  a  biomimetic  vision  sensor,  (2) 
creation  of  algorithms  to  extract  pertinent  image  features  such  as  object  edges  from  the  sensor 
data,  (3)  fabrication  and  characterization  of  sensor  prototypes,  (4)  creation  of  an  automated 
sensor  calibration  subsystem,  (5)  creation  of  a  light  adaptation  subsystem  to  permit  use  of  the 
sensor  in  both  indoor  and  outdoor  real-world  environments. 

Phase  2,  “Lightweight,  Low  Power  Robust  Means  of  Removing  Image  Jitter,”  AFRL-RX- 
TY-TR-201  1-0096-02,  develops  an  optimal  platform  stabilization  mechanism  for  motion 
detection  and  target  tracking  using  recent  advances  in  the  area  of  Parallel  Kinematic  Machines 
(PKMs).  Novel  PKM  architectures  have  been  developed  for  high  performance  disturbance 
rejection  and  target  tracking.  Nonlinear  feedback  control  has  been  successfully  implemented  on 
this  hardware  allowing  for  maximum  feedback  and  stability  despite  the  presence  of  nonlinearities 
in  the  feedback  loop.  Modified  command  feedforward  has  been  successfully  implemented  on 
these  PKMs  that  provides  high  performance  tracking  despite  significant  nonminimum  phase 
delay.  The  combination  of  these  new  PKMs  and  advanced  control  methods  is  a  substantial  new 
Air  Force  capability  for  future  unmanned  ground  vehicle  applications. 

Phase  3,  “Unification  of  Control  and  Sensing  for  More  Advanced  Situational  Awareness,” 

AFRL-RX-TY-TR-201 1-0096-03,  develops  a  multi-purpose  planning  scheme  that  effectively 
solves  patrolling  and  constrained  sensor  planning  problems  for  a  large-scale  multi-agent  system. 
This  situation  arises  when  some  mobile-robots  are  performing  a  patrolling  mission,  and,  at  the 
same  time,  they  gather  visual  data  from  pre-defined  objects  in  the  map.  The  proposed  technique 
doesn't  directly  tackle  the  merged  problem  with  the  complicated  cost  function.  Instead,  it 
suggests  a  sequential  scheme  that  considerably  simplifies  the  main  multi-objective  problem,  and 
makes  it  easier  to  make  a  compromise.  The  simulation  confirms  the  effectiveness  of  the 
proposed  multi-objective  multi-agent  planning  scheme  for  different  size  systems. 

This  report  covers  Phase  3,  “Unification  of  Control  and  Sensing  for  More  Advanced 
Situational  Awareness,”  AFRL-RX-TY -TR-20 1 1  -0096-03 . 


1 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 


2.  INTRODUCTION 


Modem  robots  are  equipped  with  redundant  sensors  and  accurate  actuators,  and  they  employ 
reliable  algorithms  to  continuously  update  their  belief  about  the  location  in  the  environment  map 
{Localization1).  They  also  exploit  powerful  path-  planning2  techniques  to  find  an  adequate  path 
to  get  to  the  desired  destination.  Although  localization  and  path-  planning  are  the  most 
fundamental  problems  to  solve  in  order  to  have  truly  autonomous  capabilities,  they  usually  are 
not  the  main  objective  of  the  robotics.  The  ultimate  goal  of  an  autonomous  robot  is  to  effectively 
accomplish  a  mission  that  might  be  composed  of  several  tasks/sub-tasks  [1], 

In  many  applications  in  robotics,  a  group  of  robots  should  collaboratively  perform  specific  tasks 
which  have  dissimilar  mathematics  representations,  i.e.  with  a  different  objective  function.  Since 
different  situations  impose  distinct  constraints  on  each  decision  maker,  a  decent  plan  of  action  is 
required  to  allocate  tasks  to  agents  in  this  cooperative  decision  making  system.  Likewise,  a 
surveillance  mission  usually  consists  of  different  tasks  such  as  patrolling,  tracking,  object 
detection,  visual  data  gathering,  etc.  Many  of  these  tasks  are  separately  studied  in  literature, 
while  the  combinations  of  those  are  very  occasionally  taken  into  consideration.  The  main  reason 
is  that  the  combined  large-scale  problem  results  in  a  very  complicated  formulation,  and  it 
demands  a  meticulous  multi-objective  planning  scheme  that  achieves  diverse  goals  at  the  same 
time.  Therefore,  making  a  reasonable  compromise  to  reach  a  satisfactory  result  for  a  large-scale 
problem  would  be  impractical. 

This  paper  proposes  a  multi-objective  planning  scheme  that  effectively  performs  Patrolling  and 
constrained  visual  sensor  management  (CVSM)  for  a  large-scale  surveillance  system.  Both 
patrolling  and  CVSM  are  emerging  research  areas  and  have  various  applications  in  different 
domains  such  as  surveillance,  environmental  monitoring,  etc.  The  proposed  planning  scheme 
first  determines  the  patrolling  trajectories  for  all  agents,  and  it  determines  which  object  should  be 
inspected  by  which  camera  at  each  time  step.  The  ultimate  objective  is  minimizing  the  maximum 
time  lag  between  passing  by  the  same  place  (revisit  time)  for  all  desired  places.  Simultaneously, 
it  is  desired  to  maximize  the  overall  quality  of  observations.  The  proposed  scheme  defines  each 
task's  parameters  in  such  a  way  that  they  can  ensure  desirable  results.  In  other  words,  it  reduces 
the  complexity  of  the  original  problem  by  dividing  the  merged  complex  problem  into  sub-tasks 
at  the  cost  of  losing  optimality.  The  proposed  multi-objective  planning  scheme  requires  a  grid- 
based  map,  target  positions,  and  patrolling  via  points  as  inputs.  The  algorithm  starts  with  the 
classic  patrolling  problem,  and  solves  the  corresponding  traveling  salesperson  problem  (TSP)  to 
design  the  best  possible  trajectories  that  minimize  patrolling  time.  Then,  it  partially  modifies  the 
trajectories  by  considering  their  effects  on  both  task's  performance,  and  adds  one  new  via  point 
to  a  trajectory  that  causes  the  minimum  increase  in  visiting  time.  In  this  process,  the  patrolling 
performance  gradually  degenerates  in  order  to  attain  an  acceptable  performance  level  for  the 
combined  scheme.  Since  the  algorithm  locally  computes  the  inspection  quality  of  the  current 
patrolling  trajectories  caused  by  a  specific  target,  it  is  computationally  fast,  and  applicable  for 
large-scale  multi-agent  systems.  After  the  path  planning  process,  a  hybrid  method  is  employed  to 
find  a  near  optimal  planning  for  the  CVSM  problem.  Results  show  that  far  fewer  observations 
are  required  to  obtain  a  specific  amount  of  visual  information  from  all  sides  of  the  targets. 


1  The  process  of  determining  the  location  of  the  robot  relative  to  the  environment  is  called  Localization. 

2  Planning  in  this  context  means  devising  a  trajectory  from  a  starting  point  to  a  desired  destination. 


2 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 


Reference  [1]  exploits  ordered  upwind  methods  to  combine  area  patrol,  perimeter  surveillance 
and  target  tracking  problems.  Machado  et  al.  (2003)  study  various  architectures  of  the  patrolling 
problem,  and  give  some  guidelines  to  choose  an  adequate  multi-agent  system  strategy  for  multi¬ 
agent  patrolling  tasks.  Using  a  specific  class  of  cyclic  strategies,  a  multi-agent  patrolling  problem 

can  be  modeled  as  the  TSP,  and  it  can  be  satisfactorily  solved  in  0(n 3)  [2],  Xiong  and  Svensson 
[3]  perform  a  comprehensive  survey  about  sensor  fusion  and  sensor  management  methods  and  its 
issues.  Tarabanis  et.  al.  [4]  exclusively  investigate  visual  sensor  management  in  the  3-D 
computer  vision  area.  In  order  maximize  the  quality  of  information,  [5]  conclusively  establishes 
entropy  based  information  metrics  to  differentiate  effective  sensors. 

This  report  is  organized  as  follows.  Section  3  begins  with  a  brief  problem  description  of  the 
Multi-Objective  Multi- Agent  system  followed  by  its  sub-tasks  (patrolling  and  CYSM)  in 
Sections  3.1  and  3.2.  Section  3.3  explains  the  proposed  coordinated  design  procedure.  Section  4 
contains  all  simulations  for  different  problem  instances.  We  conclude  in  Section  5  with 
recommendations . 
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3.  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 

The  proposed  planning  scheme  is  aimed  at  solving  a  class  of  Multi-Objective  Multi- Agent 
problems  that  can  be  represented  as  a  patrolling  and  CVSM  problem.  This  situation  occurs  in 
many  surveillance  applications  whenever  a  number  of  agents  are  patrolling  an  area,  and  collect 
information  from  some  prime  targets.  The  proposed  scheme  is  applicable  to  any  similar 
applications. 

The  theory  will  be  developed  within  the  context  of  ground  situational  awareness.  In  order  to  keep 
the  report  self-contained  as  much  as  possible,  in  the  subsections  3.1  and  3.2,  we  briefly  review 
both  constituent  problems,  and  explain  all  requisite  theoretical  aspects.  Afterwards,  Section  3.3 
elaborates  the  formulation  of  the  mixed  problem. 

3.1.  Patrolling 

Patrolling  is  the  act  of  repeatedly  traversing  a  region  in  order  to  perform  condition  monitoring 
and  detect  unexpected  changes  in  the  map  states.  There  are  two  different  approaches  to  perform 
patrolling:  deterministic  and  probabilistic.  Probabilistic  methods  are  designed  to  prevent 
adversaries  from  entering  a  specific  area  by  minimizing  the  penetration  probability.  They  employ 
randomized  strategies  to  make  the  routes  unpredictable. 

Deterministic  approaches  achieve  less  sweeping  time  for  a  group  of  agents  and  visit  all  key 
points  in  a  fixed  area.  Our  new  method  is  specifically  designed  for  the  latter  patrolling  strategy. 
However,  since  the  overlapping  trajectories  are  complex  and  unknown  to  an  adversary,  they  still 
maintain  low  penetration  probability  in  most  cases.  Because  new  trajectories  can  automatically 
be  generated  very  quickly  without  human  intervention,  they  can  be  changed  often  to  keep  the 
patrols  from  becoming  predictable. 

It  can  be  shown  that  the  deterministic  patrolling  problem  can  be  formulated  as  a  TSP  which  can 
be  effectively  solved  by  an  approximate  method  [6]. 

3.2.  Constrained  Visual  Sensor  Management 

VSM's  main  objective  is  to  specify  the  parameters  of  the  vision  system  to  obtain  worthwhile 
information  from  the  environment  [7].  The  CVSM  problem  is  a  special  case  of  VSM  in  which 
the  agents  (cameras)  and  targets  are  subject  to  some  constraints,  e.g.  they  are  moving  on  specific 
paths.  A  CVSM  problem  size  is  relatively  smaller  in  terms  of  the  number  of  unknown  variables 
than  VSM.  Consequently,  it  is  easier  to  find  an  optimal  solution  for  a  small  size  CVSM  problem. 
However,  a  large-scale  CVSM  problem  is  much  harder  to  solve  due  to  a  lot  of  linear/non-linear 
constraints.  It  can  be  shown  the  CVSM  can  be  formulated  as  an  Integer  Linear  Programming 
(ILP)  problem  [8]  which  lies  in  the  strongly  N  P  -hard  class  of  combinatorial  optimization 
problems.  Like  the  TSP  problem,  no  Polynomial  Time  Approximation  Scheme  (PTAS)  has  been 
discovered  to  find  an  optimal  solution  for  such  problems. 

Suppose  decisions  are  represented  by  U  e  BnxmxN  which  is  a  binary  control  tensor  where  n ,  m  and 
N  are  the  number  of  targets,  observers  and  samples,  respectively.  Let  uljk ,  a  binary  variable,  be 
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an  entry  of  tensor  U ,  where  uijk  =  1  indicates  that  target  i  will  be  inspected  by  Observer  j 
during  sample  k . 

Let  UJk  e  B"xl  be  a  binary  column  vector  as  Equation  1. 

Ujk  =[uiJk],  For  i  =  l,2,---,n  ^ 

Furthermore,  let  Q  represent  the  observation  quality  where  L  is  the  number  of  different  sides 
from  which  each  target  must  be  viewed.  Q  can  be  computed  using  a  priori  knowledge  of  the 
motion  dynamics  of  the  targets  and  observers  as  well  as  the  environmental  conditions  at  least  for 
a  short  period  of  time.  Each  entry  of  Q  (  qijkl )  is  a  positive  number,  and  it  denotes  the  observation 

quality  of  direction  l  of  target  i  from  observer  j  at  sample  k  (See  Section  3.2.1). 

Let  Qjk  be  the  observation  quality  matrix  for  observer  j  at  sample  k  for  all  target  in  all  different 
sides  (Equation  2). 


QJk=kijkil  For  i  =  h2,---,n  1  =  1,2,— ,L  (2) 

The  CVSM  problem  can  also  be  expressed  by  Equation  3  [8]. 

N  m 

Maximize  {min.  [£  £  diag (UJk )  Qjk  ]} 

k= 1  j= 1 

Subject  to : 

i= 1 

For  j  =  1, 2, k  =  l,2,---,N 

U  e  BnxmxN  (3) 

Nourzadeh  and  Mclnroy  [9]  propose  an  approximate  hybrid  PTAS  algorithm  (ILP-Greedy)  to 
solve  the  CVSM  problem.  The  proposed  method  can  achieve  a  near-optimal  solution  by  letting 
each  constituent  algorithm  operate  in  a  domain  where  they  perform  better. 

3.2.1.  Image  Quality  Assessment 

The  output  of  an  optical  imaging  system  depends  on  several  physical  factors,  i.e.  lighting, 
atmospheric  attenuation,  light  diffraction,  occlusion,  imaging  sensor  resolution  and  sensitivity, 
electronics  parts  and  output  devices.  In  this  surveillance  application,  some  physical  aspects  have 
a  significant  influence  on  the  output  image.  The  subsequent  subsections  develop  methods  of 
including  the  most  important  physical  aspects:  light  diffraction,  occlusion  and  side  observation 
quality. 
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3.2.1. 1.  Light  Diffraction 

Because  of  the  Fraunhofer  (far- field)  diffraction  phenomenon,  even  an  optical  lens  with  perfect 
quality  has  limited  performance,  and  by  increasing  the  object  distance,  the  resolving  ability 
decreases.  The  image  resolving  quality  for  a  system  with  a  fixed  aperture  diameter  D  and  fixed 
wavelength  A  can  be  expressed  by  Equation  4,  where  8  is  the  distance  between  the  object  and  the 
lens. 


7  resolve 


D 


\22A8 


(4) 


3.2. 1.2.  Occlusion 

Since  an  occluded  object  can't  be  inspected  by  the  observer,  the  resulting  image  quality  for  that 
object  should  be  considered  zero.  On  the  other  hand,  if  there  is  no  obstacle  between  the  object 
and  observer  the  possibility  of  having  perfect  image  of  that  object  exists  (Equation  5). 

[ Vias  ~  0  occlusion 

Ute  =  1  no  occlusion  (5  ) 

3.2. 1.3.  Side  Observation  Quality 

In  visual  sensor  planning,  the  vast  majority  of  research  is  mainly  concerned  with  finding  the 
parameters  of  the  vision  system  to  inspect  all  sides  of  an  object  with  a  minimum  number  of 
observations.  One  simple  way  to  compute  this  quality  is  to  assign  an  outward  facing  unit  normal 
for  each  face  of  interest,  i.e.  let  nilk  be  the  Ith  outward  facing  unit  normal  on  object  i  at  sample 
time  k .  To  determine  if  a  specific  side  of  an  object  is  viewed,  let  vijk  be  the  unit  vector  pointing 
from  object  /  to  observer  j  at  sample  time  & .  Then  the  observation  quality  of  side  l  of  object  i 
can  be  computed  as  in  Equation6. 


ilk!  _  \nilkVijk 
*4  view  j  q 


nMVijk  >  0 
nmvijk  ^  0 


(6) 


3.2.1 .4.  Single  Quality  Metric 

In  order  to  form  a  single  number  reflecting  overall  observation  quality  of  object  i  inspected  by 
observer  j  during  sample  k  along  face  / ,  all  calculated  qualities  should  be  combined.  Since 
overall  quality  varies  from  application  to  application,  it  can  be  done  by  multiplying  weighted 
qualities.  This  method  allows  us  to  control  the  participation  of  all  constituent  parts  in  a  single 
quality  metric.  Let  qijkl  be  the  product  of  all  mentioned  qualities  (Equation  7). 


Clijkl 


(7) 


For  different  values  ofz  ,j  and* ,  qijkl forms  a  4-D  observation  quality  array  Q . 
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3.3.  Procedure  for  Coordinated  Patrolling 


Consider  a  surveillance  application  in  which  m  mobile  robots  are  patrolling  an  area  with  some 
obstacles.  While  they  are  detecting  some  adversaries,  they  also  take  N  pictures  of  n  pre-defined 
moving/fixed  targets  in  a  2-D  grid-based  map  of  the  area  within  a  certain  period  of  time.  It  is 
desirable  to  have  decent  quality  images  from  different  sides  of  the  targets.  Now  the  question  is 
which  paths  are  the  best  to  perform  both  tasks  efficiently.  To  answer  this  question,  the  patrolling 
and  CVSM  problems  should  be  merged  in  a  way  so  they  could  find  some  patrolling  trajectories 
for  robots  on  which  the  patrolling  time  is  minimized  and  Equation  3  is  maximized. 

In  other  words,  the  TSP  problem  intends  to  find  the  shortest  path  between  key  points,  while  the 
CVSM  tries  to  maximize  the  minimum  observation  qualities  of  targets.  Like  many  Multi¬ 
objective  optimization  problems,  in  this  design  procedure  there  is  a  tradeoff  between  two  cost 
functions. 

For  the  sake  of  simplicity,  suppose  that  the  targets  are  fixed.  In  this  case,  the  observation  quality 
of  direction  /  of  target  i  from  all  points  p  in  the  map  forms  a  scalar  quality  field,  and  the  total 
effect  of  this  quality  field  along  curve  (trajectory)  ccan  be  expressed  by  the  line  integral  in 
Equation  8.  This  line  integral  can  be  rewritten  by  Equation  9, 


[j~j  Qi  (Xj )  d,s 

(8) 

h 

\Qi(xR(t),xi)dt 

(9) 

h 


where  xt  and  xR  denote  state  vectors  of  target  i  and  robot  j ,  respectively.  In  the  discrete  case,  the 
latter  integral  will  be  quality  summation  over  a  finite  number  of  points  on  all  the  trajectories. 
[Qsumh  ]  is  a  matrix  representing  observation  quality  for  all  sides  and  targets(Equation  10). 

c  N 

Qsumu  =  ZZaw(tv() 

k= 1  (10) 

Figure  1  depicts  some  possible  solutions  for  one  instance  of  this  mixed  problem.  Blue  circles  and 
red  stars  represent  key  patrolling  points  and  targets,  respectively.  Evidently,  the  solid-line 
trajectories  are  not  optimal  with  respect  to  the  CVSM  problem's  objective  function.  For  the 
robots  on  these  trajectories,  it  is  impossible  to  inspect  the  sides  of  targets  in  arrows  directions. 
For  instance,  all  robots  which  are  moving  on  path  Cj  and  C2  can  see  only  one  side  of  targets  1, 4 
and 3 .  Therefore,  the  minimum  inspection  quality  would  be  zero. 
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Figure  1.  Multi-objective  Multi-agent  System  (Surveillance  Application  for  Mobile  Robots) 


On  the  other  hand,  none  of  the  dotted  trajectories  is  optimal  in  the  sense  of  sweeping  distance. 
But  they  make  a  satisfactory  compromise  for  the  Multi-Objective  problem.  The  prime  goal  of 
this  paper  is  to  devise  a  PTAS  algorithm  that  finds  a  near  optimal  solution  for  the  combined 
problem. 

3.3.1.  Proposed  Multi-Objective  Planning  Algorithms 

Algorithm  1  states  the  proposed  planning  scheme  to  coordinated  design  of  the  Multi-Objective 
Multi-agent  problem.  The  algorithm  begins  with  solving  the  patrolling  problem  for  some  pre¬ 
defined  via  points.  Any  algorithm  that  solves  the  deterministic  patrolling  problem  can  be  used  to 
achieve  the  patrolling  trajectories.  In  this  paper,  the  A  path  finding  algorithm  is  used  to  find  the 
minimum  distances  between  the  patrolling  via  points.  For  dynamic  maps,  one  can  utilize  any  un¬ 
informed  incremental  search  algorithm  such  as  D  ,  etc.  After  finding  distances  between  via 
points,  the  TSP  is  solved  to  find  the  best  patrolling  trajectories.  The  resulting  patrolling 
trajectories  minimize  the  revisit  time  of  via  points.  Any  existing  approximation  scheme  can  be 
exploited  to  achieve  the  patrolling  paths. 

In  line  2  of  the  algorithm,  matrix  Qsum  is  computed  according  to  Equation  10.  This  matrix  tells 
how  these  trajectories  contribute  in  all  targets  observation  qualities  from  different  sides. 
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Algorithm  1  Coordinated  Design  of  the  patrolling  and  the  CVSM  problems 


Input:  Map  information,  targets  states  x- ,  Patrolling  via  Points  Ppts ,  robots  states  xjr 

Output:  New  patrolling  trajectories  and  CVSM  decision  tensor  U 

1  Use  a  TSP  approximation  to  solve  the  patrolling  problem  and  find  patrolling  trajectories 

2  Compute  the  Qsum  matrix 

3  Find  the  minimum  element(s)  ( J min )  of  Qsum 

4  Find  the  corresponding  target  and  side  indices  of  Jmin  in  Qsum - >  ( tldx ,  sldx ) 

Compute  the  observation  quality  of  side  sldx  of  target  tldx  from  free  cells  in  the  vicinity  of  target 
tldx . 

Among  k  -largest  qualities,  choose  cell  C..  that  has  minimum  distance  from  one  of  the  patrolling 
6  J 
trajectories 

7  Add  Cy  to  the  patrolling  via  points  set  Ppts 

8  Solve  the  patrolling  problem  just  for  updated  Trajectory.  Update  Qsum  matrix 

9  If  Jmin<tr  then 

10  |  Goto  3; 

11  End 


The  algorithm  continues  to  find  a  worst  inspection  quality  among  all  targets.  Then  it  locally 
computes  observation  qualities  of  that  particular  side  of  the  selected  target  from  its  neighboring 
free  cells.  All  quality  calculations  are  performed  locally,  which  makes  the  proposed  algorithm 
fast  and  efficient. 

Among  the  k  -largest  quality  values,  a  cell  that  has  the  minimum  distance  from  the  patrolling 
trajectories  is  chosen.  Now,  it  forms  a  new  patrolling  trajectory  by  adding  this  point  to  the  list  of 
via  points  of  the  nearest  patrolling  trajectory.  The  algorithm  resumes  by  solving  the  patrolling 
problem  for  the  updated  trajectory,  and  updates  the  Qsum  matrix.  If  this  improvement  is 
satisfactory,  the  multi-agent  sensor  management  algorithm  is  performed  for  the  obtained 
trajectories. 

Otherwise  it  keeps  changing  the  trajectories  until  the  minimum  observation  quality  passes 
threshold  tr .  The  threshold  tr  is  a  design  parameter  used  to  find  an  acceptable  patrolling/CVSM 
tradeoff.  The  difficulty  with  this  parameter  is  that  there  is  no  general  rule  to  tune  tr ,  and  the  tr 
value  will  be  different  from  application  to  application.  Thus  a  lot  of  trial  and  error  might  be 
required  to  find  a  good  tradeoff  for  the  problem  at  hand. 
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4.  RESULTS  AND  DISCUSSION 


Figure  2  illustrates  a  planning  problem  in  which  three  groups  of  mobile-robots  are  patrolling  an 
area  in  three  different  trajectories.  The  patrolling  via  points  on  patrolling  trajectories  1,  2  and  3 
are  shown  by  diamond,  pentagram  and  hexagram  markers,  respectively.  While  patrolling,  they 
gather  visual  information  from  ten  different  targets  (blue  arrows)  in  a  2-D  grid-based  map. 

Figure  3  and  Figure  4  depict  the  quality  field  caused  by  targets  in  different  sides.  In  these  graphs, 
observers  can  take  best  quality  pictures  in  the  red  areas.  Suppose  in  this  surveillance  mission,  the 
minimum  required  observation  quality  for  all  sides  of  the  targets  is  0.3  ( tr  =  0.3  ). 

Table  1  shows  the  computed  Qsum  matrix  on  these  patrolling  paths.  In  this  configuration,  since 
robots  on  these  trajectories  cannot  inspect  at  least  a  side  of  some  targets  (side  1  of  the  target  9 
and  side  2  of  the  target  1),  the  minimum  inspection  quality  would  be  zero.  The  side  observation 
qualities  less  than  threshold  level  are  shown  in  red. 


Table  1.  Sides  Observation  Quality  of  the  Targets  on  the  Patrolling  Trajectories  (Qsum) 


Side 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target  1 

s 

1 

2 

3 

4 

5 

6 

7 

8 

9 

0 

1 

6.46 

8.56 

0.04 

2.20 

8.68 

4.35 

1.53 

0.14 

0.00 

17.48 

2 

0.00 

0.19 

8.42 

1.74 

4.01 

0.10 

7.00 

10.42 

4.38 

5.29 
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Figure  2.  Patrolling  Trajectories  before  Performing  Planning 


11 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 


Quality  distribution  of  side  1  Quality  distribution  of  side  2 


The  quality  distribution  of  the  sum  of  the  sides  qualities 


Figure  4.  Sum  of  the  Observation  Quality  Fields  of  the  Targets  in  Different  Sides 


Figure  5  illustrates  the  system  configuration  after  running  one  step  of  the  proposed  Multi- 
Objecting  Muli-agent  planning  scheme.  Note  that  one  via  point  is  added  to  the  patrolling 
trajectory  1.  By  looking  at  Figure  3  ,  it  is  obvious  that  a  via  point  has  been  added  in  an  area  in 
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map  with  high  quality  level  of  the  side  2.  Since  this  cell  has  the  minimum  distance  to  trajectory 
1,  it  will  be  added  to  via  points  of  trajectory  1,  and  forms  a  new  patrolling  trajectory.  Table  3 
shows  the  Qsum  matrix  after  performing  one  iteration  of  Algorithm  1 .  Not  only  is  the 
observation  quality  of  Target  1  improved  but  also  on  this  new  patrolling  trajectory  side  2  of 
target  6  is  observable  with  a  quality  level  higher  than  the  threshold  tr  =  0.3  . 

Figures  6-8,  depict  the  changes  made  by  the  proposed  algorithm  in  three  successive  iterations. 
Tables  4-6  show  the  corresponding  change  to  the  Qsum  matrix. 


Table  2.  Performance  of  the  Proposed  Muli 


i-ob  jective  Multi-agent  Planning  Scheme 


Iteration 

Revisit  time  (seconds) 

Revisit  time  percentage  increase 

^ mm 

Trajectory  1 

Trajectory  2 

Trajectory  3 

Trajectory  1 

Trajectory  2 

Trajectory  3 

0 

15.546 

11.389 

36.627 

0.000 

0.000 

0.000 

0.0 

1 

17.278 

11.389 

36.627 

11.143 

0.000 

0.000 

0.0 

2 

17.278 

15.010 

36.627 

11.143 

31.796 

0.000 

0.036 

3 

17.278 

15.718 

36.627 

11.143 

38.005 

0.000 

0.138 

4 

17.278 

15.718 

37.042 

11.143 

38.005 

1.131 

1.209 

Performance  of  the  patrolling  and  minimum  achievable  observation  quality  for  all  targets  are 
outlined  in  Table  2.  After  performing  four  iterations  of  Algorithm  1,  the  revisit  time  percentage 
increases  for  the  patrolling  trajectories  1,  2  and3  are  11.143,  38.005  and  1.131  respectively.  This 
increase  in  revisit  time  is  necessary  to  improve  the  visual  inspection  quality  (Jmin).  As 
mentioned  before,  the  threshold  level  tr  can  significantly  change  the  performance  of  both 
patrolling  and  CVSM.  Like  many  Multi-objective  optimization  problems,  proposing  a  general 
rule  to  design  the  threshold  level  is  not  straightforward  and  it  varies  from  application  to 
application. 


Table  3.  Sides  Observation  Quality  of  the  Targets  after  Performing  One  Step  of  the 

Proposed  Algorithm  ( Qsum ) 


Side 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target  1 

s 

1 

2 

3 

4 

5 

6 

7 

8 

9 

0 

1 

6.16 

9.83 

0.04 

5.77 

11.80 

6.21 

1.53 

0.14 

0.00 

13.14 

2 

2.08 

0.19 

9.29 

4.98 

7.52 

1.23 

7.78 

10.62 

5.14 

5.38 
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Figure  5.  Patrolling  Trajectories  after  Performing  One  Step  of  the  Proposed  Planning 


Table  4.  Sides  Observation  Quality  of  the  Targets  after  Performing  Two  Steps  of  the 

Proposed  Algorithm  ( Qsum ) 


Side 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target  1 

s 

1 

2 

3 

4 

5 

6 

7 

8 

9 

0 

1 

6.20 

11.16 

0.04 

5.77 

9.89 

6.20 

2.18 

0.14 

1.56 

13.18 

2 

2.08 

0.19 

9.83 

4.93 

7.99 

1.23 

11.16 

11.25 

10.27 

5.43 
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Figure  6.  Patrolling  Trajectories  after  Performing  Two  Steps  of  the  Proposed  Planning 


Table  5.  Sides  Observation  Quality  of  the  Targets  after  Performing  Three  Steps  of  the 

Proposed  Algorithm  ( Qsum ) 


Side 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target  1 

s 

1 

2 

3 

4 

5 

6 

7 

8 

9 

0 

1 

6.10 

6.83 

3.02 

5.77 

8.15 

6.20 

4.93 

0.14 

2.40 

13.17 

2 

2.09 

3.90 

7.73 

4.78 

7.22 

1.21 

6.77 

10.46 

7.82 

4.96 
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Figure  7.  Patrolling  Trajectories  after  Performing  Three  Steps  of  the  Proposed  Planning 


Table  6.  Sides  Observation  Quality  of  the  Targets  after  Performing  Four  Steps  of  the 

Proposed  Algorithm  ( Qsum ) 


Side 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target 

Target  1 

s 

1 

2 

3 

4 

5 

6 

7 

8 

9 

0 

1 

6.03 

6.49 

3.02 

5.77 

7.81 

6.19 

4.72 

3.31 

2.40 

13.04 

2 

2.09 

3.84 

7.48 

4.75 

7.22 

1.21 

6.74 

9.86 

7.70 

4.74 
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Figure  8.  Patrolling  Trajectories  after  Performing  Four  Steps  of  the  Proposed  Planning 


The  next  step  after  reaching  an  acceptable  level  of  the  observation  quality  for  each  target  is  to 
solve  a  CVSM  problem  for  some  moving  camera  (on  patrolling  trajectories)  and  some 
moving/fixed  targets  given  their  relative  motion.  Figure  9  shows  the  corresponding  CVSM 
problem  results  based  on  a  method  proposed  in  [9].  Using  this  hybrid  optimization  method  (ILP- 
Greedy),  a  near  optimal  planning  scheme  is  found  in  polynomial  time.  Figure  10  depicts  one 
snapshot  of  this  planning  in  which  six  mobile  robots  are  patrolling  an  area  and  also  gathering 
some  visual  information  from  the  targets. 
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5.  CONCLUSIONS  AND  RECOMMENDATIONS 


In  this  report,  a  method  is  developed  to  determine  observation  planning  for  some  moving  objects 
and  observers  (cameras)  in  which  the  motion  dynamics  and  the  environmental  conditions  are 
known.  The  simulation  results  are  very  promising.  The  length  of  trajectories  are  effectively 
changed  by  adding  one  more  patrolling  via  points  to  each  patrolling  trajectory.  This  allows  both 
objectives  (patrolling  and  visual  inspection)  to  be  accomplished. 

Even  with  a  few  number  of  observers,  the  proposed  scheme  designs  new  patrolling  trajectories 
on  which  a  high  level  of  visual  measurement  quality  is  achievable.  We  recommend  the  use  of 
this  new  algorithm  as  a  method  to  unify  control  and  sensing  and  thereby  enhance  situational 
awareness. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 


CVSM 

constrained  visual  sensor  management 

ILP 

integer  linear  program 

NP 

non-polynomial 

PTAS 

polynomial  time  approximation  scheme 

TSP 

traveling  salesman  problem 

VSM 

visual  sensor  management 
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